Alveoli, teeth, and tooth loss: Understanding the homology of internal mandibular structures in mysticete cetaceans

The evolution of filter feeding in baleen whales (Mysticeti) facilitated a wide range of ecological diversity and extreme gigantism. The innovation of filter feeding evolved in a shift from a mineralized upper and lower dentition in stem mysticetes to keratinous baleen plates that hang only from the roof of the mouth in extant species, which are all edentulous as adults. While all extant mysticetes are born with a mandible lacking a specialized feeding structure (i.e., baleen), the bony surface retains small foramina with elongated sulci that often merge together in what has been termed the alveolar gutter. Because mysticete embryos develop tooth buds that resorb in utero, these foramina have been interpreted as homologous to tooth alveoli in other mammals. Here, we test this homology by creating 3D models of the internal mandibular morphology from terrestrial artiodactyls and fossil and extant cetaceans, including stem cetaceans, odontocetes and mysticetes. We demonstrate that dorsal foramina on the mandible communicate with the mandibular canal via smaller canals, which we explain within the context of known mechanical models of bone resorption. We suggest that these dorsal foramina represent distinct branches of the inferior alveolar nerve (or artery), rather than alveoli homologous with those of other mammals. As a functional explanation, we propose that these branches provide sensation to the dorsal margin of the mandible to facilitate placement and occlusion of the baleen plates during filer feeding.